Modeling Tumor/Polyp/Lesion Structure in 3D for Computer-Aided Diagnosis in Colonoscopy

We describe a software system for building three-dimensional (3D) models from colonoscopic videos. The system is end-to-end in the sense that it takes as input raw image frames—shot during a colon exam—and produces the 3D structure of objects of interest (OOI), such as tumors, polyps, and lesions. We use the structure-from-motion (SfM) approach in computer vision which analyzes an image sequence in which camera’s position and aim vary relative to the OOI. The varying pose of the camera relative to the OOI induces the motion-parallax effect which allows 3D depth of the OOI to be inferred. Unlike the traditional SfM system pipeline, our software system contains many check-and-balance mechanisms to ensure robustness, and the analysis from earlier stages of the pipeline is used to guide the later processing stages to better handle challenging medical data. The constructed 3D models allow the pathology (growth and change in both structure and appearance) to be monitored over time.